Decision-Theoretic Planning

ions and hierarchical approaches While the envelope extension method ignores portions of the state space, other techniques have considered abstractions of the state space that try to group together sets of states that behave similarly under the chosen actions of the optimal policy. Boutilier and Dearden (Boutilier & Dearden 1994) assume a representation for actions that is similar to that of Buridan (Kushmerick, Hanks, & Weld 1994) described earlier and a state utility function that is described in terms of domain literals. They then pick a subset of the literals that accounts for the greatest variation in the state utility and use the action representation to find literals that can directly or indirectly affect the chosen set, using a technique similar to the one developed by Knoblock for building abstraction hierarchies for classical planners (Knoblock 1991). This subset of literals then forms the basis for an abstract MDP by projection of the original states. Since the state space size is exponential in the set of literals, this reduction can lead to considerable time savings over the original MDP. Boutilier and Dearden prove bounds on the difference in value of the abstract policy compared with an optimal policy in the original MDP. Dean and Lin refine this idea by splitting the MDP into subsets and allowing a different abstraction of the states to be considered in each one (Dean & Lin 1995). This approach can provide a better approximation because typically different literals may be relevant in different parts of the state space. However there is an added cost to re-combining the separate pieces unless they happen to decompose very cleanly. Dean and Lin assume the partition of the state space is given by some external oracle. Factored action and state representations While the last approaches exploited structure in the state utility description it is also possible to exploit it in the action description. Boutilier et al. extend modified policy iteration to propose a technique called structured policy iteration that makes use of a structured action representation in the form of 2-stage Bayesian networks (Boutilier, Dearden, & Goldszmidt 1995). The representation of the policy and utility functions are also structured, using decision trees. In standard policy iteration, the value of the candidate policy is computed on each iteration by solving a system of jSj linear equations (step 2 in Table 2), which is computationally prohibitive for large real-world planning problems. Modified policy iteration replaces this step with an iterative approximation of